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This brief provides a more detailed examination and 
illustration of a particular issue, disproportionality, 
which can be situated within the framework. We begin 
this brief by reviewing the framework with respect to 
its application to address disproportionality. This is 
followed by an illustration of specific analytic steps, 
and concludes by providing guidance for further 
inquiry that links back to the guiding principles in the 
framework. 


In addition to the guidance found in both briefs, 
Guidance for Examining District Alternate Assessment 
Participation Rates (Evans & Domaleski, 2018) and 

this brief, Guidance for Examining Disproportionality of 
Student Group Participation in Alternate Assessments, 
NCEO and the National Center for the Improvement 
of Educational Assessment (NCIEA) produced a third 
companion resource, a video training module: Guidance 
for Examining Participation Rates and Disproportionality 
(Evans & Domaleski, 2019). The video training module 
covers the principles and process necessary to engage 
in areview of participation and disproportionality 

on the AA-AAAS. It also provides some examples to 
demonstrate analytic techniques in Excel with sample 
data. The video may be accessed at: 


https://nceo.info/Resources/videos. 


Examining disproportionality with regard to alternate 
assessment participation is essentially an inquiry into 
whether certain groups are over- or under-identified 
as having a significant cognitive disability. This brief 

is intended to fill a gap created by the lack of an 
explicit definition or guidance on how to calculate 
disproportionality for students with significant 
cognitive disabilities who participate in an AA-AAAS. 
We distinguish this requirement for the alternate 
assessment from the definition and requirements 
under the Individuals with Disabilities Education Act 
(IDEA) that requires states to collect and examine data 
to determine whether significant disproportionality 

is occurring in the overall population of students 
receiving special education services (IDEA Section 
618(d)). 


Specifically, we address the following key questions: 


e What is disproportionality related to the waiver 
requirements for AA-AAAS? 


e What are some appropriate analytical approaches 
for monitoring disproportionality in student 
groups that take an AA-AAAS under federal law? 


e What guidance can improve the practice of 
understanding, calculating, interpreting, and 
monitoring disproportionality in student groups 
that take an AA-AAAS? 


ESSA Requirements 


The Every Student Succeeds Act (ESSA) of 2015 
requires states to apply for a waiver prior to the 
testing window if they think they will go over the 

1% participation rate cap for students with the 

most significant cognitive disabilities taking an 
AA-AAAS (34 CFR 200.6(c)(2)). Part of a state’s 
waiver application is verifying and addressing 
disproportionality in the identification of students 
with the most significant cognitive disabilities. 
Specifically, the state must provide evidence that it has 
verified that each district? that the state anticipates 
will test more than 1% of its assessed students in any 
subject using an AA-AAAS: 


(1) followed the state’s guidelines for participation in 
the AA-AAAS; and 


(2) will address any disproportionality in the 
percentage of students in any subgroup under section 
1111(c)(2)(A), (B), or (D) of the Act taking an AA-AAAS 
(34 CFR 200.6(d)), consistent with section 612(a)(16) 
(C) of the IDEA. 


These student groups include seven racial and ethnic 
groups (White, Black or African American, Hispanic, 
Native American or Alaska Native, Asian, Pacific 
Islander, and Multiracial), socio-economic status (as 
determined by a students’ eligibility for Free and 
Reduced Price Meals), and English learners. The 
state must also provide a plan and timeline with clear, 
actionable steps and milestones for how the state will 


1Term used synonymously with Local Education Agency (LEA). 


address any disproportionality in the percentage of 
students taking an AA-AAAS as identified through the 
data provided by districts (34 CFR 200.6(c)(4)(iv)). 


As stated previously, the law does not explicitly define 
the term “disproportionality” or provide any guidance 
on methodology for analysis of disproportionality for 
students with the most significant cognitive disabilities 
taking an AA-AAAS. 


What is Disproportionality? 


Disproportionality exists when there are atypical 
differences in the proportions of participants froma 
student group who take the alternate assessment in 
comparison to the general assessment. 


A comprehensive investigation of disproportionality 
involves the use of both analytical techniques and 
other sources of evidence to corroborate findings 
whenever possible. We distinguish our definition of 
disproportionality from requirements under IDEA 
that require states to collect and examine data to 
determine whether significant disproportionality is 
occurring in the overall special education population 
with respect to the identification of students with 
disabilities, the placement of such students in 
particular educational settings, and the incidence, 
duration, and type of disciplinary actions of such 
students (IDEA Section 618(d)). 


Overall Framework 


The previous NCEO brief (Evans & Domaleski, 2018) 
provided an overall framework for analyzing district 
alternate assessment participation rates. We start by 
highlighting some key points from that framework that 
are broadly applicable to examining disproportionality 
in alternate assessment participation. The 

framework consists of four guiding principles that 
characterize approaches likely to be more effective 

in identifying ‘exceptional’ versus ‘expected’ district 
alternate assessment participation rates and also 
disproportionality in the identification of certain 


groups of students with the most significant cognitive 
disabilities who take an AA-AAAS. 


1. A comprehensive solution cannot be purely 
empirical due to small n-sizes. 

Analyses and inferences are challenged by small 
n-sizes when examining overall district participation 
rates in alternate assessments. The issue with 

small n-sizes is exacerbated in this context because 
those small n-sizes are now to be broken down 
(disaggregated) for student groups. For this reason, 
it is important to be cautious about the analyses 
conducted and the inferences that can be made at 
each level. For example, an analysis that drills all the 
way down to student groups at the school level is 
going to significantly limit the claims or inferences in 
which one can have confidence. Moreover, minimum 
cell sizes and overall n-sizes are extremely important 
to consider in this situation for public reporting 
purposes to prevent individual students from being 
identified from their unique personal characteristics. 


It is important for states to streamline and prioritize 
the targeted groups for their analyses given the 
constraints provided by small n-sizes and associated 
uncertainty. States should understand the limits of 
expanding the number of groups included, especially 
at certain levels (e.g., district, school). This indicates 
the need for careful examination of the data and use 
of multiple sources of evidence when attempting to 
determine disproportionality of certain groups taking 
an AA-AAAS. States should combine judicious data 
analyses with thoughtful inquiry that is tailored to the 
unique context of the state and districts in the state, 
and that also takes into account multiple years of data 
and other sources of evidence whenever possible. 


2. It is important to detect atypical or “exceptional” 
values. 

This principle draws our attention to the need to 
develop a baseline for expectations. In our earlier 
NCEO brief, we highlighted several research-based 
assumptions that can help detect when certain values 
are ‘expected’ versus ‘exceptional’ and therefore in 
need of additional scrutiny. We can employ a similar 
procedure here, although there is no evidence to 


suggest that the student groups required under 
federal law (race/ethnicity, free-and-reduced price 
lunch (FRL) status, English learner (EL) status) are 
more or less likely to have the most significant 
cognitive disabilities and take an AA-AAAS. We might 
assume that there should be similar proportions 
of students with significant cognitive disabilities 
from across race/ethnicity categories, FRL status, 
and EL status compared to the general population. 
The question is at what level should we make this 
assumption? 


Given the even smaller n-sizes for group analyses it 

is likely advisable to set a baseline expectation at the 
highest unit of analysis possible with multiple years of 
data. This is especially important for smaller districts 
or districts with relatively few students in one or more 
groups. In the case of very small n-sizes, even multi- 
year analyses may not be sufficient to produce an 
appropriate n-size for an analysis that can reasonably 
support an inference about disproportionality. In such 
instances, it is particularly important to corroborate 
findings with other sources of information, such as 
data at the state level, data from other (especially 
comparable) districts, conversations with district 
special education administrators, etc. 


When examining data for disproportionality in 
certain student groups taking an AA-AAAS, states 
should consider the potential for an interaction to 
occur between the under- or over-identification of 
certain groups (e.g., students from certain racial/ 
ethnic groups with disabilities) with an examination of 
disproportionality. The under-identification of some 
student groups has been documented in the overall 
special education population (e.g., Morgan, Farkas, 

& Hillemeier, 2017), but not as well established in 
the population of students with the most significant 
cognitive disabilities. Given this lack of research 
evidence about the “true” distribution of students 

by group who take an alternate assessment, these 
errors can potentially create misleading assumptions 
about expected participation rates and confound 
interpretation of results. 


3. Apply methods to address uncertainty. 

As stated in the previous NCEO brief (Evans & 
Domaleski, 2018), uncertainty is a reality for any 
inferential statistical analyses where judgments are 
made about the likelihood that an observed difference 
might have happened by chance alone. Uncertainty 

is acritical issue for the AA-AAAS due to the even 
smaller n-sizes that occur during subgroup analyses. 
Methods discussed in the earlier brief included 
confidence intervals and multiple years of data. In the 
next section, we discuss these methods applied to AA- 
AAAS disproportionality calculations. 


4. The culminating decision and subsequent actions 
based on the evidence is (a) a matter of degree, and 
(b) related to unique context and circumstances. 
Making an error in one direction or the other can 

have real implications for students with the most 
significant cognitive disabilities. Type | error (false 
positive: an unwarranted claim of disproportionality) 
and Type II error (false negative: failure to detect 
disproportionality) apply here. Districts identified with 
‘exceptional’ or atypical values in disproportionality 
for certain groups on an AA-AAAS will have to provide 
aclear description of not only how they followed the 
state’s guidelines for participation in the AA-AAAS, 
but also how the district will address the identified 
disproportionality. The consequence of a Type | error 
results in expenditure of time and resources that may 
be unwarranted. The consequence of a Type II error is 
a failure to detect and address a problem of possible 
inequitable practice. 


Due to the extremely small n-sizes for certain 
subgroups, there is likely no single piece of evidence 
sufficient to decide with certainty about whether 

any identified disproportionality is warranted. We 
suggest evaluating the preponderance and collection 
of evidence in light of state and district trends over 
multiple years of data to describe a level of confidence 
along a continuum. 


Analytic Approaches to Explore 
Disproportionality 


Two analytic approaches from among a range of 
potentially broader approaches are recommended 
here to examine disproportionality in AA-AAAS 
participation. The central purpose of these analyses 
is to quantify the difference between observation 
and expectation. More specifically, the questions of 
interest are: 


e Isthe proportion of students who participate 
in the AA-AAAS consistent with the proportion 
expected to participate? 


e Dothe differences between participation 
and expectation vary across student group 
membership? 


e Are the differences sufficiently pronounced to 
suggest the finding is meaningful? 


To address these questions, we propose the following 
process. 


Step 3: 
Determine if 
difference is 
meaningful 


Step 1: 


Establish participation 
rate for each focal group 


Step 2: 
Calculate test 
statistic 


Step 1: Establish Participation Rate for Each Focal 
Group 

Focal group refers to any student group that is being 
examined, such as students who are economically 
disadvantaged, English learners, students ina 
particular race/ethnic group, etc. 


Establishing the participation rate for each focal 
group (i.e., the student group under analyses) is 
confounded by the problem of small n-sizes. One 
approach to address the small n-size challenge is to 
produce the rate based on multiple years of data. For 
example, Table 1 illustrates how three years of data 


may be combined to create a multi-year calculation 
for AA-AAAS participation for one group. 


As illustrated in Table 1, the annual participation rates 
varied for the focal group from 28.7% to 36.9% over 
the period examined. The multi-year participation 
rate for the focal group during the period is 32.9%. 
This is simply calculated as the sum of focal group 
participants divided by the sum of all participants 
(148/450). 


Step 2: Calculate the Test Statistic 

Here we illustrate how to calculate a test statistic 

to quantify the difference between the AA- 

AAAS observed participation rate and expected 
participation rate. We illustrate this using two 
approaches: (a) difference in proportions, and (b) risk 
ratios. 


Difference in proportions. The difference in 
proportions is simply the observed proportion 

of focal group AA-AAAS participants minus the 
“expected proportion.’ Although there are many ways 
to establish the “expected proportion,’ we suggest 
simply using the proportion of focal group members 
in the population for the unit of analysis (e.g., district). 
We present each proportion as a percent for ease of 
interpretation. For example, consider Table 2, which 
illustrates hypothetical participation rates for a 
district over multiple years. 


The focal group proportion is calculated in the same 
manner as illustrated in Table 1: the number of 
focal group AA-AAAS participants is divided by the 
total AA-AAAS participants, yielding a participation 
rate or proportion of 32.9% (148/450). Similarly, 
the proportion for the non-AA-AAAS students is 
calculated by dividing the focal group students who 
are not AA-AAAS participants by the total number 


Table 1. Illustration of Calculating Participation Rates with Multi-Year Data 


po 2016 2017 | 2018 | Three Year Total 


Total AA-AAAS participants | 150 | 143 | 57 
= a a 


150 
Number of focal group participants 49 
Number of non-focal group participants | 101 | 02s | 8 


143 
41 
102 


7 [450 
s [us| 


Percent of focal group participants 


32.7% 28.7% 36.9% 32.9% 


Table 2. Illustration of AA-AAAS and Non-AA-AAAS Proportions 


50,000 


Focal Group Proportions (%) 


of non-AA-AAAS participants (15,000/50,000)—a 
proportion of 30.0%. 


The difference in proportions is produced by simply 
subtracting the focal group proportion for AA-AAAS 
participants from the proportion of focal group 
students who are not AA-AAAS participants (32.9% 

- 30.0%), which is a difference of 2.9%. In other 
words, there are about 3% more focal group students 
participating in the AA-AAAS than one would expect 
based on the proportion of focal group students in the 
general population. This procedure can be repeated 
for all focal groups of interest. 


Risk ratio. Another way to describe the relationship 
of observed versus expected proportions is with a risk 
ratio, sometimes termed “relative risk.” This is simply 
calculated by dividing the proportion of focal AA- 
AAAS participants by the proportion of focal group 
students who are non-participants. Drawing from the 
previous example, the calculation is as follows: 


Proportion of Focal Group Participants 32.9 


= 1.10 
Proportion of Focal Group NonParticipants 30.0 


Arisk ratio of 1.0 indicates that the proportion of focal 
group students who are AA-AAAS participants is the 
same as the proportion of focal group students who 
are not AA-AAAS participants. In other words, there is 
no evidence of disproportionality when the risk ratio 
is at or very near 1. Values greater than 1 indicate that 
the probability or “risk” of a focal group student being 
an AA-AAAS participant is greater than expectation. 
For example, a risk ratio of 2.0 indicates that a student 
who is amember of the focal group is twice as likely 

to participate in the AA-AAAS. Values less than 1.0 
indicate that a focal group student is less likely to be 
an AA-AAAS participant than expected. For example, 
arisk ratio of 0.5 indicates that students in the focal 


30.0% 


group are half as likely to be AA-AAAS participants. 


Step 3: Determine if the Difference is Meaningful 

The final and perhaps most challenging step is to 

make a determination of whether the difference in 
proportion or risk ratio is meaningful. That is, do the 
results provide evidence to support or refute a claim of 
disproportionality? 


One way to address this question is by conducting 

a test of statistical significance on the test statistic 
computed in step 2. This is commonly accomplished by 
computing a confidence interval and then evaluating 
whether the test statistic is outside of the interval for 
a desired level of confidence (e.g., 95%). Confidence 
intervals can be produced for differences in 
proportions or risk ratios, the latter of which requires 
a transformation to a natural log. The technical details 
for producing these confidence intervals are beyond 
the scope of this paper (see, for example, Agresti, 1996 
for details). However, the video module, produced as 

a companion to this paper, provides more technical 
details and illustrative examples. 


If we assume a 95% confidence interval on the 
proportion of focal group students participating in 

an AA-AAAS in the previous example is +/- 4.34, the 
“true range” for the proportion would be from 28.6% 
to 37.2%; this takes into account variance due to the 
n-size. This range includes the 30% expectation, based 
on the proportion of focal group students who are not 
AA-AAAS participants. This would not be statistical 
evidence of disproportionality (or that the proportions 
are significantly different). 


It is important to remember that statistical tests are 
always an incomplete inquiry into ‘meaningfulness, 
particularly in the present context given the challenge 
of low n-sizes. We caution against making aclaim 


about disproportionality based strictly on significance 
testing. Whether a finding is considered meaningful 
should be informed by inquiries such as: 


e Are there pronounced differences between the 
results in the current year compared to previous 
years? 


e Are there distinct differences for one or more focal 
groups compared to results from other entities 
(e.g., results from one district are very different 
compared to other demographically-similar 
districts in the current or prior year)? 


e Aretheresults consistent with available literature/ 
research base? 


e Arethere contextual factors that should be taken 
into account (e.g., a spike in enrollment of students 
with significant cognitive disabilities due to 
expansion of services provided)? 


Ultimately, we refer back to the principles in the 
framework to guide decision making about findings. 
The decision should take into account qualitative as 
well as quantitative information; multiple sources 

of information should be evaluated to address 
uncertainty; and context should be considered. Finally, 
decisions are typically a matter of degree and require a 
blend of evidence with informed judgment. 


Conclusion 


The broad framework and general principles outlined 
in an earlier NCEO brief (Evans & Domaleski, 2018) 
can be applied to disproportionality and used by states 
to ensure that any examination of disproportionality 
for a student group participating in an AA-AAAS takes 
into account the unique challenges associated with 
disaggregating already small n-sizes. Due to the small 
n-size of the alternate assessment student population, 


it is difficult (if not impossible) to set solely empirical 
guidelines for when a district’s alternate assessment 
participation rate is disproportionate. We recommend 
a measured approach that combines appropriate 
quantitative analyses and qualitative inquiry at the 
highest unit of analysis possible (e.g., district or state), 
uses multiple years of data to increase the n-size, and 
makes judicious decisions about the selection of focal 
student groups. 


We also urge caution when examining 
disproportionality in student groups at the district 
level, particularly when the total district alternate 
assessment population size is small. Differences in 
student group participation rates based on small 
n-size may appear as large differences in proportions 
or relative risk ratios. It is advisable to consider the 
differences in proportions or risk ratios alongside 
frequency counts to inform a better judgment about 
disproportionality and associated certainty. 
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